This invention relates to an adaptive control system for a media encoder.
In real-time video telephony, dynamic and high system loads at a transmitting device supporting a video encoder can lead to degradation of video quality at the receiving device due to overload of the video encoder. This can be observed in the received video stream as increased end-to-end delays, delay spikes, frame freezes and jerky play-out. It is therefore important to ensure that the demands made of a video encoder are well-matched to its capabilities. This is generally straightforward to achieve for any given device, but presents a significant challenge when the video encoder might be supported at a wide range of different devices.
In order to estimate the capabilities of a video encoder as embodied at a given device, a combination of two different types of metrics is usually employed. Static metrics provide a measure of the theoretical performance of the architecture in which the video encoder is supported. For a software encoder, static metrics might include maximum processor frequency, system bus frequency, and the number of processor cores in the system. However, such metrics do not provide an accurate measure of the abilities of an encoder at a given point in time.
Dynamic metrics can help to improve the estimate of the capabilities of a video encoder by introducing measurements of the actual performance of the underlying device. For a software encoder, dynamic metrics might include process and thread execution times, measures of the idle time of a processor, and current processor frequency. However, such dynamic metrics suffer from a variety of problems. For multi-threaded encoders, measurements of execution times are unreliable due to context switching. And in systems having multiple cores, average measurements of idle time can be unreliable because some cores might be lightly loaded while others are overloaded.
In modern systems, the problems of using existing metrics as estimates of the load on a software video encoder can be even more acute. Multi-threading, virtualization and the use of logical processors all pose further problems to the use of static metrics. Often systems also make use of voltage and frequency scaling, as well as dynamically switching on and off processor cores, to improve energy efficiency, particularly in portable devices.
It can be more difficult to estimate the load on a hardware video encoder because the capabilities of the hardware video encoder are often largely independent of the capabilities of the device at which it is provided. For example, while a CPU of the device is largely idling, the hardware encoder might be fully loaded. In devices that make use of hardware encoders in the encoder pipeline the use of conventional static and dynamic metrics to estimate encoder load can be wholly inaccurate.
As well as leading to poor adaptation of an encoder to its real-time capabilities, the use of existing metrics as a measure of encoder load make it difficult to accurately perform dynamic calibration of a video encoder. This is because conventional calibration mechanisms generally require some measure of encoder load in order to judge when the processing capabilities of the encoder are exceeded.
The above issues are not limited to video encoders, with other types of media encoder facing similar problems.